I need to make this also support caching calls to whole workflows. If I use the files in the output tasks' caches instead of fetching the cached results of one workflow, I get different paths (that symlink to the same files), and then I can't re-use MiniWDL cached calls that depend on files from the called workflow.

I think we might run into problems with sibling files with WDL call caching when reading caches written by MinIWDL. MiniWDL will lay out task outputs by output name, and then reference those locations in the cache files. So files that were siblings won't be anymore when loaded from MiniWDL's task output directories.

I think we might just have to let that not work. When Toil writes files into the cache to reference them it should obey the sibling files constraint because it uses the same code as when writing them for a task to pick paths.

https://ucsc-ci.com/databiosphere/toil/-/jobs/79122 got a Cannot assign requested address errors trying to do an HTTP download in src/toil/test/wdl/wdltoil_test.py::WDLTests::test_miniwdl_self_test_by_reference. I don't think it has anything to do with that test specifically, except that it needs to do networking. We might have a problem with network connectivity on the Gitlab runners in addition to the problem that led us to assign a static IP to the main Gitlab server.

@stxue1 Can you maybe look at this? I tried to merge my code that tacks a shared filesystem path onto every File that has one with your code that defers virtualization to task boundaries, but I know I'm missing at least one route that files can come in, because I don't think I am managing to add the shared filesystem path when a workflow-level string is coerced to a file and then virtualized. Where is that happening, and at that point do we have a good way to distinguish a leader-filesystem file from a worker-filesystem file?

Also, I went through and turned all the setattr file mutations and file.value = mutations into immutable replacements of the file with the version that has the attribute set/has its value set, because I was not confident that I understood the implications of changing the original File object. Is this approach going to work?

@stxue1 Can you maybe look at this? I tried to merge my code that tacks a shared filesystem path onto every File that has one with your code that defers virtualization to task boundaries, but I know I'm missing at least one route that files can come in, because I don't think I am managing to add the shared filesystem path when a workflow-level string is coerced to a file and then virtualized. Where is that happening, and at that point do we have a good way to distinguish a leader-filesystem file from a worker-filesystem file?

The virtualization for when a workflow-level string being coerced to a file takes place in two areas. On task boundaries, virtualization is handled when virtualize_files() is called manually. The other place is when _read/_write is called, which happens when a string value is used implicitly in a standard library function that accepts files. Both ultimately point back to the _virtualize_file function, so I believe for optimal coverage caching should be right before this function is called.

I think the two places that matter is in the virtualize_filescall at the task boundary in WDLTaskWrapperJob (I think a file can be coerced from a string here, but I will have to double check). The other place that would matter is in the _read and _write calls in the standard library. I think it would be the _read call in ToilWDLStdLibBase and the _write call in ToilWDLStdLibTaskCommand, though my immediate hunch is that the TaskCommand stdlib shouldn't be touched as that is more for miniwdl internally (I left a docstring for the reasoning why they differ, if it isn't clear I can come up with a better docstring). It seems like the new ToilWDLStdLibWorkflow._write does properly handle the virtualization and adds the caching system along with it. I think the function to check if caching is needed/supported is the ToilWDLStdLibBase, which still seems to be used in certain areas.

Also, I went through and turned all the setattr file mutations and file.value = mutations into immutable replacements of the file with the version that has the attribute set/has its value set, because I was not confident that I understood the implications of changing the original File object. Is this approach going to work?

The mutable replacements should be fine, the metadata they copy over includes the necessary information for virtualization.

at that point do we have a good way to distinguish a leader-filesystem file from a worker-filesystem file?

I suppose there isn't currently a very good way to distinguish if the file.value is pointing to a worker filesystem file or a leader-filesystem file. Any use of a file will automatically try and read from the virtualized value, creating it if it doesn't exist (and thus uploading it to the jobstore), then changing the file.value field to the correlated devirtualized path. This behavior is left over from before when we read virtualized values that were set in the file.value field. Perhaps we should change it so the file.value field always points to a path that exists on the associated worker/leader.

My main concern is figuring out where exactly to do this without breaking path substitution support. In theory, I think at the beginning of every job should be fine.

Steven's File Facts

The original string value is usually stored as the File value, unless the File is virtualized and sent to a task command job, and the task tries to use it, since there inside the task it will have a value pointing to a copy from the file store. If the file is passed through the task at the WDL level, when it gets back to the workflow scope it has the original string in the value.

In the task wrapper the original string path is used.

JSON-based inputs get stored as file:/usr/lib/whatever but string coercions get stored as /usr/lib/whatever.

Strings coerced inside task output sections appear as toilfile: URIs at workflow scope.